Search engines use a process named web crawling where they use crawlers like, bots and spiders to visit a page or download a page. This process has been used to periodically visit the pages to examine if there is any change in its contents. The robot files of a webpage are essential to perform the web crawling. The name “crawler” comes from the technique of the process, where the program crawls through a page at a time and then follow the links of the other pages of that site, until all the pages have been examined (Rouse, 2005).